[SPARK-57834][SQL] Support nanosecond-precision timestamps in the sequence function - #58708
Open
stevomitric wants to merge 4 commits into
Open
[SPARK-57834][SQL] Support nanosecond-precision timestamps in the sequence function#58708stevomitric wants to merge 4 commits into
stevomitric wants to merge 4 commits into
Conversation
…uence function ### What changes were proposed in this pull request? Extend the `sequence(start, stop, step)` function to accept nanosecond-precision timestamp endpoints (`TIMESTAMP_LTZ(p)` / `TIMESTAMP_NTZ(p)`, p in 7..9), producing an array of the same nanosecond type. A nanosecond value is carried as a `TimestampNanosVal` object rather than a primitive `long`, so the existing primitive-array sequence machinery cannot hold it directly. Instead the microsecond sequence logic is reused on `epochMicros` (keeping all of the existing DST, month, overflow and length handling) and each generated element is re-wrapped as `TimestampNanosVal.fromParts(micros, startFraction)`, materialized through `GenericArrayData`. Membership is decided on the microsecond grid (every step is microsecond-granular) and each element keeps the start value's sub-microsecond fraction. The `SEQUENCE_WRONG_INPUT_TYPES` error message now lists the nanosecond timestamp types among the accepted start/stop types. ### Why are the changes needed? Part of the nanosecond-precision timestamp umbrella (SPARK-56822). Without this, `sequence` rejected nanosecond timestamp endpoints even though the microsecond timestamp types were supported. ### Does this PR introduce any user-facing change? Yes. `sequence` now accepts nanosecond-precision timestamp start/stop values and returns an array of that type. ### How was this patch tested? New unit test in `CollectionExpressionsSuite` (LTZ and NTZ, precisions 7/8/9, day-time and default and negative and single-element and null cases; interpreted and codegen) and updated `SEQUENCE_WRONG_INPUT_TYPES` assertions in `DataFrameFunctionsSuite`. Co-authored-by: Isaac <no-reply@databricks.com>
Follow-up: drop the explanatory comment in the nanosecond case of Sequence.impl. Co-authored-by: Isaac <no-reply@databricks.com>
…uence Follow-up to code review on the nanosecond-precision sequence support. - Correctness: the sequence used to run membership on epochMicros only, which let a result overshoot `stop` when `start` had a larger sub-microsecond fraction than `stop`, and suppressed the "illegal sequence boundaries" error when the endpoints shared a microsecond but the step pointed the wrong way. The bound and the default-step sign now use the full-precision value: `nanosStepIsNegative` and `nanosBoundedStopMicros` nudge the delegated microsecond bound off the boundary microsecond according to the endpoints' fractions, so the result never exceeds `stop` and an out-of-order same-microsecond pair still raises the boundary error. - Tests: add differing-fraction cases (overshoot excluded, same-microsecond boundary error), day-time (Duration) and year-month (Period) interval steps, and an end-to-end SQL test in TimestampNanosFunctionsSuiteBase; fix a comment that mislabeled a calendar interval as a day-time interval step. - Cleanups: collapse the identical timestamp/date/nanos arms in checkInputDataTypes into one case; unify the nanosecond `impl` branch with the microsecond one via a nanosecond-to-microsecond type map; use fromTrustedRowBytes to skip re-validating the loop-invariant fraction on every element. Co-authored-by: Isaac <no-reply@databricks.com>
…p signs Broaden the nanosecond sequence tests to cover the sub-microsecond boundary for both step directions and the equal-microsecond default-step case: the review-cited example (stop on a whole second, larger start fraction), a negative step where an element on stop's microsecond is dropped vs kept, and equal-microsecond endpoints with no explicit step (the default step direction follows the full-precision comparison). Test-only; the full-precision bound was already implemented. Co-authored-by: Isaac <no-reply@databricks.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Extend the
sequence(start, stop, step)function to accept nanosecond-precision timestamp endpoints (TIMESTAMP_LTZ(p)/TIMESTAMP_NTZ(p), p in 7..9), producing an array of the same nanosecond type.A nanosecond value is carried as a
TimestampNanosValobject rather than a primitivelong, so the existing primitive-array sequence machinery cannot hold it directly. Instead the microsecond sequence logic is reused onepochMicros(keeping all of the existing DST, month, overflow and length handling) and each generated element is re-wrapped asTimestampNanosVal.fromParts(micros, startFraction), materialized throughGenericArrayData. Membership is decided on the microsecond grid (every step is microsecond-granular) and each element keeps the start value's sub-microsecond fraction.The
SEQUENCE_WRONG_INPUT_TYPESerror message now lists the nanosecond timestamp types among the accepted start/stop types.Why are the changes needed?
Part of the nanosecond-precision timestamp umbrella (SPARK-56822). Without this,
sequencerejected nanosecond timestamp endpoints even though the microsecond timestamp types were supported.Does this PR introduce any user-facing change?
Yes.
sequencenow accepts nanosecond-precision timestamp start/stop values and returns an array of that type.How was this patch tested?
New unit test in
CollectionExpressionsSuite(LTZ and NTZ, precisions 7/8/9, day-time and default and negative and single-element and null cases; interpreted and codegen) and updatedSEQUENCE_WRONG_INPUT_TYPESassertions inDataFrameFunctionsSuite.Was this patch authored or co-authored using generative AI tooling?
Co-authored-by: Claude Opus 4.8